For more than a decade, there have been video file servers using an admissions control policy, task scheduling, caching, and pre-fetching for delivering isochronous streams of video data for viewing of the video data in real time. See, for example, Vishlitzky et al. U.S. Pat. No. 5,737,747 issued Apr. 7, 1998, and Vahalia et al. U.S. Pat. No. 5,933,603, issued Aug. 3, 1999. Such a video file server includes a cached disk array storage subsystem and a plurality of stream server computers linking the cached disk storage array to a data network for the transfer of video data streams. The cached disk storage subsystem is responsive to video pre-fetch commands, and the data specified for a pre-fetch command for a process is retained in an allocated portion of the cache memory from the time that the cached disk storage subsystem has responded to the pre-fetch command to the time that the cached disk storage subsystem responds to a fetch command specifying the data for the process. The time between pre-fetching and fetching is selected based on available disk and cache resources. The video file server provides video-on-demand service by maintaining and dynamically allocating sliding windows of video data in the random access memories of the stream server computers.
More recently the introduction of high definition (HD) video has resulted in a significant increase in “dropped frames” when HD video has been streamed from video file servers originally designed streaming standard definition (SD) video. At lower SD bit-rates the problem can be solved by using caching mechanisms in the storage arrays to hide any possible high latency I/Os that can result in dropped frames when no buffering is used. On the other hand pre-fetch mechanisms like read-ahead and write-behind can hide high disk latencies up to a certain limit, e.g. up to a certain bit rate most probably not high enough for uncompressed HD streams running at 150-200 MB/sec.
The problem becomes more difficult in cases where different video streams are fed concurrently to multiple host applications from same storage array. In this latest case the read-ahead and pre-fetch mechanisms of the storage array cache are less efficient as the access to the storage becomes random and the disk seek times increase resulting in an increased latency and a higher probability to have long disk response times. A cache trashing effect occurs when the storage array has to throw or discard cached data as a result of cache misses, causing the storage access latencies to increase and more frames to be dropped. In short, the storage array can still deliver the required HD throughputs, but cannot guarantee the instantaneous throughput required, and the result is that frames are dropped as decoders try to compensate for higher response times than the duration of decoding a frame.
Attempts to reduce dropped frames when delivering HD throughputs have included using more cache memory in the video server or in the video decoders for further read ahead or write-behind to compensate for higher storage latencies, and the use of very large File System blocks increasing with sequential access. If this caching is sufficient to ensure smooth playout, however, there are large latencies up to 500 msec.
The most advanced video servers include a metadata server in a shared storage architecture. The metadata server allocates the data blocks and builds bitmaps and manages metadata of a shared file system. Each host computer directly accessing the storage array obtains access rights and metadata from the metadata server using a File Mapping Protocol (FMP). See, for example, Fridella et al. U.S. Patent Application Pub. 2007/0055702 A1 published Mar. 8, 2007, incorporated herein by reference, for details regarding the File Mapping Protocol, and Faibish et al. U.S. Patent Application Publication 2007/0260842 A1 published Nov. 8, 2007, incorporated herein by reference, for details regarding layout of the file system on disk to enhance I/O performance.
The most advanced video servers, however, are challenged by new video applications, such as rendering farms, which may consume multiple streams of uncompressed HD video at rates higher than the usual rate for human viewing. Rendering farms are now synthesizing video animations and virtual environments for diverse applications such as cartoon and movie making for entertainment and advertising, accident reconstruction for litigation support, simulations for industrial design and engineering, and architectural rendering for virtual property tours.